Multi-layer neural networks using symmetric tensors

ABSTRACT

Methods and apparatuses for implementing a neural network using symmetric tensors. In embodiments, a system may include a higher order neural network with a plurality of layers that includes an input layer, one or more hidden layers, and an output layer. Each of the input layer, the one or more hidden layers, and the output layer includes a plurality of neurons, where the plurality of neurons includes at least first order neurons and second order neurons, and where inputs at a second order neuron are combined using a symmetric tensor.

TECHNICAL FIELD

The present disclosure generally relates to the field of neuralnetworks, in particular, to multi-layer neural networks using symmetrictensors.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Unless otherwiseindicated herein, the materials described in this section are not priorart to the claims in this application and are not admitted to be priorart by inclusion in this section.

Current Neural Network (NN) topologies used for deep learning includefully connected layers, typically as output layers. Typically, neuronsused in the fully connected layer represent hyperplanes, or in otherwords, multi-dimensional linear units. To resolve non-linear problems acombination of several layers of linear units may be used.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will be readily understood by the following detaileddescription in conjunction with the accompanying drawings. To facilitatethis description, like reference numerals designate like structuralelements. Embodiments are illustrated by way of example, and not by wayof limitation, in the figures of the accompanying drawings.

FIG. 1 illustrates an overview of an example system having a neuralnetwork, in accordance with various embodiments.

FIG. 2 illustrates an example neural network, in accordance with variousembodiments.

FIG. 3 illustrates an overview of an example process of using the neuralnetwork system, in accordance with various embodiments.

FIG. 4 illustrates an example implementation of a quadratic symmetrictensor versus a linear hyperplane, in accordance with variousembodiments.

FIG. 5 illustrates an example exclusive or (XOR) problem implementationas a symmetric tensor versus a linear hyperplane, in accordance withvarious embodiments.

FIG. 6 illustrates an example implementation of a single neuron in ahidden layer based on a distribution of training patterns, in accordancewith various embodiments.

FIG. 7 illustrates an example implementation of 12 neurons in a hiddenlayer based on a distribution of training patterns, in accordance withvarious embodiments.

FIG. 8 illustrates an example computer system, suitable for use topractice the present disclosure (or aspects thereof), in accordance withvarious embodiments.

FIG. 9 illustrates an example storage medium with instructionsconfigured to enable a computer system to practice the presentdisclosure, in accordance with various embodiments.

FIG. 10 illustrates an example implementation related to classificationproblems, in accordance with various embodiments.

DETAILED DESCRIPTION

The present disclosure presents methods and apparatuses for high orderNNs with multiple layers, where the neurons include at least secondorder neurons. Embodiments may include a system comprising a higherorder neural network having a plurality of layers including an inputlayer, one or more hidden layers, and an output layer, where each of theinput layer, the one or more hidden layers, and the output layerincludes a plurality of neurons. The plurality of neurons may include atleast first order neurons and second order neurons where the inputs at asecond order neuron may be combined using a symmetric tensor.

In order to resolve non-linear problems, neurons in legacyimplementations may combine several layers of linear units, increasingthe memory required and the computer complexity for evaluation andtraining. Existing higher order NNs typically contain only one hiddenlayer, and do not take advantage of deep learning approaches. As aresult, they may have training rules are typically computationallyexpensive.

Embodiments described herein may increase the order of the NN anddecrease the number of layers in the NN. This may result in a reductionof the number of activation functions required and may reduce the timeconsumed for training. For example, each layer removed may reducehundreds of activation functions.

Embodiments of high-order units may be used to solve non-linear problemsusing non-linear hyperplanes, which may reduce the number of neuronsused in a NN and may also reduce the number of layers. This may reducethe number of transcendental functions by reducing the need for layers.As a result, this may reduce the training and computational complexityotherwise required by using legacy linear units.

Embodiments as disclosed herein may increase the order of the NN byusing a symmetrical tensor, reducing the number of layers and unitsrequired to resolve a non-linear problem and having a substantial impacton the complexity of evaluation and training.

In the description to follow, reference is made to the accompanyingdrawings, which form a part hereof wherein like numerals designate likeparts throughout, and in which is shown by way of illustrationembodiments that may be practiced. It is to be understood that otherembodiments may be utilized and structural or logical changes may bemade without departing from the scope of the present disclosure.Therefore, the following detailed description is not to be taken in alimiting sense, and the scope of embodiments is defined by the appendedclaims and their equivalents.

Operations of various methods may be described as multiple discreteactions or operations in turn, in a manner that is most helpful inunderstanding the claimed subject matter. However, the order ofdescription should not be construed as to imply that these operationsare necessarily order dependent. In particular, these operations may notbe performed in the order of presentation. Operations described may beperformed in a different order than the described embodiments. Variousadditional operations may be performed and/or described operations maybe omitted, split or combined in additional embodiments.

For the purposes of the present disclosure, the phrase “A and/or B”means (A), (B), or (A and B). For the purposes of the presentdisclosure, the phrase “A, B, and/or C” means (A), (B), (C), (A and B),(A and C), (B and C), or (A, B and C).

The description may use the phrases “in an embodiment,” or “inembodiments,” which may each refer to one or more of the same ordifferent embodiments. Furthermore, the terms “comprising,” “including,”“having,” and the like, as used with respect to embodiments of thepresent disclosure, are synonymous.

As used hereinafter, including the claims, the terms “interface” and“engine” may refer to, be part of, or include an Application SpecificIntegrated Circuit (ASIC), an electronic circuit, a programmablecombinational logic circuit (e.g., field programmable gate arrays(FPGA)), a processor (shared or dedicated) and/or memory (shared ordedicated) that execute a plurality of programming instructions of oneor more software or firmware programs to provide the describedfunctionality.

FIG. 1 illustrates an overview of an example system having a neuralnetwork, in accordance with various embodiments. Diagram 100 shows, inembodiments, system 102 that may include a neural network 104, which maybe connected to a communication interface 106. The system 102 may be apart of a computing system, an apparatus, or a device. Neural network104 may be used to classify given pattern or data set into predefinedclasses, trained to produce a prediction in response to a set of inputs,or identify a special feature of the input data and classify/clusterthem into different categories without any prior knowledge of the data.

In embodiments, the neural network 104 may be implemented using amultilayer approach that may involve multiple layers, at least first andsecond order neurons, where the second order neurons operate asdescribed below. In embodiments, the neural network may receive datafrom or communicate data through a communications interface 106. Thecommunications interface 106 may interact with remote or cloud basedservers 110 or data equipment 112, such as sensors or other equipmentthat may report data to be used in training or accessing the neuralnetwork 104. In embodiments, the communications interface 106 maycommunicate with a user 114 to directly or indirectly access the neuralnetwork 104. In embodiments, neural network 104 may be initially trainedby a machine learning system. Further, neural network 104 may beconfigured to adapt via self-learning, during usage.

FIG. 2 illustrates an example neural network, in accordance with variousembodiments. Diagram 200 shows a neural network 204, which may besimilar to neural network 104 of FIG. 1. The neural network 204 mayinclude an input layer 216, a plurality of hidden layers 218 (one shownfor ease of illustration and understanding), and an output layer 220.The input layer 216 may receive data x_(i) as inputs. Each hiddenlayer(s) 218 may include a number of second order neurons 224 to processthe inputs X_(i), and eventually, output layer 220 may output datay_(i). Neurons 224 include at least first order and second orderneurons.

In embodiments, a second order neuron 224 having inputs that may becombined using a symmetric tensor. In embodiments, the symmetric tensormay include a symmetric matrix Q where the inputs X_(i) are combined asfollows:S=X _(i) ^(T) QX _(i)

where X_(i) are the inputs, X_(i) ^(T) is the transpose of X_(i) and Sis the result of the combined inputs. This function may represent aquadric, which in a two-dimensional scenario may represent an ellipse ora hyperbola, a parabola, or a plane, depending upon the values of theweights represented in the matrix Q.

Quadratic Tensor.

An example of a two dimensional symmetric tensor may be a quadratictensor. In the example of a quadratic tensor, a one-dimensional inputmay be used. In this example, the symmetric tensor may be written as a2×2 matrix using one input and a bias. Generally, a new column and a newrow may be added for each additional input.

In this example, a quadratic solution using three weights as shownbelow: a, b, c.

${{\begin{bmatrix}x & 1\end{bmatrix}\begin{bmatrix}a & \frac{b}{2} \\\frac{b}{2} & c\end{bmatrix}}\begin{bmatrix}x \\1\end{bmatrix}} = {{ax}^{2} + {bx} + c}$

Then, the increment of inputs produces an increment in the number ofweights in the tensor, Ax²+2Bxy+Cy²+2Dx+2Ey+F=0

Defining the symmetric Q tensor as:

$x = {{\begin{bmatrix}x \\y \\1\end{bmatrix}\mspace{14mu} x^{T}} = {{\left\lbrack {x,y,1} \right\rbrack\mspace{14mu} Q} = \begin{bmatrix}A & B & D \\B & C & E \\D & E & F\end{bmatrix}}}$

The equation can be expressed as:x ^(T) Qx=0

Which is a quadratic equation in projective geometry.

Quadratic Tensor Training Rules.

In the quadratic tensor example, the output layer may be created usingone quadratic perceptron, which may be similar to a neuron. Inembodiments, training may be done following the gradient descendantalgorithm, and using as activation function the sigmoid,

$f = \frac{1}{\left( {1 + e^{- S}} \right)}$

where s=x^(T)Qx is the quadratic combination of all inputs, i.e. thefirst layer's outputs, and where f may be the activation function, and smay be the accumulation of energy sum. In embodiments, s may be similarto the sum of w_(i)*x_(i). In embodiments, the matrix Q may bedetermined by the weights:

$Q = \begin{bmatrix}w_{11} & w_{12} & w_{13} \\w_{12} & w_{22} & w_{23} \\w_{13} & w_{23} & w_{33}\end{bmatrix}$

In embodiments, to use the gradient descendant algorithm, the errorfunction may be defined as:

$E = {\frac{1}{2}\left( {d - f} \right)^{2}}$

Where d represents the desired output for a given input (x, y) in caseof a two dimensional input, and f represents the actual output. Thetraining rule for the i-the weight (w_(i)) may be given by the errorderivative (using the descendant gradient):

$\frac{\partial E}{\partial w_{ij}} = {\left( {d - f} \right)(f)\left( {1 - f} \right)\frac{\partial s}{\partial w_{ij}}}$

Where the derivative

$\frac{\partial s}{\partial w_{ij}} = {x_{i}x_{j}}$may have as a training rule:

$\frac{\partial E}{\partial w_{ij}} = {\left( {d - f} \right)(f)\left( {1 - f} \right)x_{i}x_{j}}$

This may represent only one extra multiplication compared with thetraditional linear perceptron. The sensibility, which may be similar tothe amount of error back-propagated to a specific input, may be computedusing:

$\frac{\partial E}{\partial x} = {\left( {d - f} \right)\left( {1 - f} \right){fQx}}$

This may produce a vector, which i-th component represent a derivativewith respect to x_(i)

Regarding each weighting factor, w_(t) may be computed as follows:w _(t) =w _(t-1)+(d−f)(f)(1−f)x _(i) x _(j)

Cubic Tensor.

Another example of a symmetric tensor may be a cubic tensor. In thisexample, the cubic tensor can be seen as a volume of weights. A 2×2×2tensor may be used for one input and tensor:

${{\begin{bmatrix}x & 1\end{bmatrix}\left\lbrack {\begin{matrix}\begin{bmatrix}x & 1\end{bmatrix} \\\begin{bmatrix}x & 1\end{bmatrix}\end{matrix}\begin{matrix}\begin{bmatrix}a & \frac{b}{3} \\\frac{b}{3} & \frac{c}{3}\end{bmatrix} \\\begin{bmatrix}\frac{b}{3} & \frac{c}{3} \\\frac{c}{3} & d\end{bmatrix}\end{matrix}\begin{matrix}\begin{bmatrix}x \\1\end{bmatrix} \\\begin{bmatrix}x \\1\end{bmatrix}\end{matrix}} \right\rbrack}\begin{bmatrix}x \\1\end{bmatrix}} = {{ax}^{3} + {bx}^{2} + {cx} + d}$

In this example, the energy collected by the inputs may be equal to:s=Q _(ijk) x _(k) x _(j) x _(i)

Cubic Tensor Training Rule.

The cubic tensor training rule may be:

$\frac{\partial E}{\partial w_{ijk}} = {\left( {d - f} \right)(f)\left( {1 - f} \right)x_{i}x_{j}x_{k}}$

With this training rule, three cumulus of energy, which may be a clusteror cloud of points in a dimensional space, may be clustered together byusing a single neuron. For example, with respect to FIG. 6, the solutionas shown by diagram 652 may be represented by a single high-order neuronusing the cubic tensor. In legacy implementations, at least two layersof perceptions with at least seven linear neurons may be required.

FIG. 3 illustrates an overview of an example process of using the neuralnetwork system, in accordance with various embodiments. As shown,process 300 shows a process that may include operations of blocks302-306. The operations may be performed by the neural network 204 ofFIG. 2. In alternate embodiments, process 300 may include more or lessoperations, or with some of the operations combined or sub-divided.

Process 300 may start at block 302. At block 302, the process may be todetermine inputs for a second order neuron in a layer of a multi-layerneural network. In embodiments, the second order neuron may correspondto neuron 224 within the hidden layer 218 of FIG. 2. The determinedinputs for the neuron may be as shown neuron 224 that may originate fromneurons in the input layer 216. In embodiments, the inputs maycorrespond to a vector X_(i), and may be from nodes within anotherhidden layer (not shown). In embodiments, inputs may represent a vectordescriptor of a pattern. Inputs may also be Mel-frequency cepstralcoefficients (MFCCs) from audio, a raw image arranged as a vector, abinarized contour from segmentation, a descriptor coming from previousconvolutional layers, and the like. In embodiments, the input maygenerally be a signature to identify a pattern that provides sufficientinformation to distinguish from other patterns.

At block 304, the process may be to combine the inputs using a symmetrictensor. In embodiments, the symmetric tensor may be as described forFIG. 2, and may include the symmetric matrix Q. In embodiments, theX_(i) input vector received from block 302 may have its transpositionmultiplied by the matrix Q, and have that result multiplied by theoriginal X_(i) input vector to determine a result.

At block 306, the process may be to output the result. In embodiments,the result may be output as input to a subsequent node of a hidden layer218 or of an output layer 220. In embodiments, the output may be theclassification result of the input or input pattern. For example, ingesture recognition the output may represent the gesture detected. Inaudio, the output may be the command or the recognized word. In imagerecognition, the output may be a representation of the imagesegmentation pixel by pixel.

FIG. 4 illustrates an example implementation of a quadratic symmetrictensor versus a linear hyperplane, in accordance with variousembodiments. Diagram 400 shows a number of points on a xy axisrepresenting input and output values. The ellipse 430 identifies acluster of points marked by a dot that represents a quadratic neuron,using a two dimensional scenario with a grade 2 tensor, of the formS═X_(i) ^(T)QX_(i). The neuron represented by the ellipse 430 largelyidentifies the desired points (represented by dots, in contrast to thosepoints represented by an x). In legacy implementations, the neuron maybe represented by a linear equation 432, which does not identify thedesired points. Rather, additional linear equations (not shown) andmultiple calculations would be need to be added as multiple neurons inlegacy implementations to linearly identify the region of the desiredpoints.

FIG. 5 illustrates an example exclusive or (XOR) problem implementationas a symmetric tensor versus a linear hyperplane, in accordance withvarious embodiments. Diagram 500 shows various stages of implementing anXOR problem using a using a symmetric tensor implemented as a quadraticneuron in comparison with a linear hyperplane. Diagram 536 shows acluster of a first set of points 538 and a second set of points 540 usedto generate (train) the XOR problem. Diagram 540 shows graphically theresult of the XOR problem shown in diagram 536 solved using legacyimplementations involving three linear units (perceptrons). Band 540 arepresents the region where points similar to points 540 may lie, andthe area 540 b outside of band 540 a represents the region where pointssimilar to points 538 may lie.

Diagram 542 shows graphically the result of the XOR problem shown indiagram 536 solved using embodiments herein, in particular a symmetrictensor implemented as a single quadratic neuron. In this case, thequadratic neuron represents a parabolic-like shape. Area 542 arepresents the regions where points similar to points 540 may lie, andarea 542 b represents the regions where points similar to points 538 maylie. In addition to providing a more accurate representation of theinitial cluster of patterns in diagram 536, the quadratic neuron alsoindicates an overlap of regions at center point 544 to indicate areas ofambiguity.

FIG. 6 illustrates an example implementation of a single neuron in ahidden layer based on a distribution of training patterns, in accordancewith various embodiments. Diagram 646 shows a first cluster of points648 and a second cluster of points 650 used to train a symmetric cubictensor as described above of a single neuron. Diagram 652 shows theresulting solution created by the single neuron, with areas 654representing regions where points similar to points 650 may lie, andwith area 656 representing regions where points similar to points 648may lie.

FIG. 7 illustrates an example implementation of 12 neurons in a hiddenlayer based on a distribution of training patterns, in accordance withvarious embodiments. Diagram 760 shows five different clusters of points762 a-762 e used to train neurons using a symmetric tensor. Diagram 764shows the resulting solution created by the 12 neurons, with areas 766a-766 e representing regions, respectively, where points similar to thepoints 762 a-762 e may lie.

FIG. 8 illustrates an example computer system, suitable for use topractice the present disclosure (or aspects thereof), in accordance withvarious embodiments.

As shown, in embodiments, computer device 800 may include one or moreprocessors 802 and system memory 804. Each processor 802 may include oneor more processor cores wherein one or more neuron of a layer of amulti-layer neural network may operate in one of the one or more cores.In embodiments, the processors 802 may be graphics processors. Inembodiments, the processors 802 may execute a plurality of threads,wherein one or more neurons of a layer of a multi-layer neural networkmay operate in one of the plurality of execution threads. Inembodiments, one or more processors 802 may include one or more hardwareaccelerators 803 (such as, FPGA), wherein at least a portion of a neuralnetwork, e.g., second order neurons may operate in the hardwareaccelerators 803. System memory 804 may include any known volatile ornon-volatile memory. Additionally, computer device 800 may include massstorage device(s) 806 (such as solid state drives), input/output deviceinterface 808 (to interface with e.g., cameras, sensors, etc.) andcommunication interfaces 810 (such as serial interface, near fieldcommunication, network interface cards, modems and so forth). Theelements may be coupled to each other via system bus 812, which mayrepresent one or more buses. In the case of multiple buses, they may bebridged by one or more bus bridges (not shown).

Each of these elements may perform its conventional functions known inthe art. In particular, system memory 804 and mass storage device(s) 806may be employed to store a working copy and a permanent copy of theexecutable code of the programming instructions in computational logic822 implementing the operations described earlier, e.g., but are notlimited to, operations associated with the NN 104 of FIG. 1 or of NNs orcomponents associated with FIGS. 2-7. The programming instructions maycomprise assembler instructions supported by processor(s) 802 orhigh-level languages, such as, for example, C, that can be compiled intosuch instructions. In embodiments, some of the functions, e.g., neuralnetwork 204 may be realized with hardware accelerator 803 instead.

The permanent copy of the executable code of the programminginstructions and/or the bit streams to configure hardware accelerator803 may be placed into permanent mass storage device(s) 806 or hardwareaccelerator 803 in the factory, or in the field, through, for example, adistribution medium (not shown), such as a compact disc (CD), or throughcommunication interface 810 (from a distribution server (not shown)).

FIG. 9 illustrates an example storage medium with instructionsconfigured to enable a computer system to practice the presentdisclosure, in accordance with various embodiments.

As illustrated, non-transitory computer-readable storage medium 902 mayinclude the executable code of a number of programming instructions 904.Executable code of programming instructions 904 may be configured toenable a system, e.g., computer device 800 of FIG. 8 or system 102 ofFIG. 1, in response to execution of the executable code/programminginstructions, to perform, e.g., various operations associated withhigher order multilayer neural networks using homogeneous symmetrictensors. In alternate embodiments, executable code/programminginstructions 904 may be disposed on multiple non-transitorycomputer-readable storage medium 902 instead. In still otherembodiments, executable code/programming instructions 904 may be encodedin transitory computer readable medium, such as signals.

In embodiments, a processor may be packaged together with acomputer-readable storage medium having some or all of executable codeof programming instructions 904 configured to practice all or selectedones of the operations earlier described. For one embodiment, aprocessor may be packaged together with such executable code 904 to forma System in Package (SiP). For one embodiment, a processor may beintegrated on the same die with a computer-readable storage mediumhaving such executable code 904. For one embodiment, a processor may bepackaged together with a computer-readable storage medium having suchexecutable code 904 to form a System on Chip (SoC).

FIG. 10 illustrates an example implementation related to classificationproblems, in accordance with various embodiments. Diagrams 1000 a-1000 fshow various stages of the semantic segmentation that may be applied tolip-reading detection, gesture recognition, face recognition, and thelike. Diagram 1000 a may represent a RGB image of a face used as inputused to train a NN having higher order nodes, such as the nodes 224within hidden layer 218 of FIG. 2 or node relating to ellipse 430 ofFIG. 4.

For example, the RGB image 1000 a may be used as input to train thedifferent hidden layers within the NN that include high-order neurons.These hidden layers may be used internally by the NN for backgroundremoval 1000 b, lip detection 1000 c, skin detection 1000 d, and/oreyes-nose-lips segmentation 1000 e. In embodiments, the combinations ofall layers may result in the semantic segmentation of face features 1000f During operation of the NN an input, such as an input of an RGB image,may result in a classification of the input image (e.g. skin detection,lip detection, etc.).

It is to be appreciated that any NN may be implemented using embodimentsdisclosed herein. As discussed above, because a high-order neuron may beimplemented using quadric equations instead of a series of vectorweights representing a hyperplane, significantly fewer high-orderneurons may be needed to implement a NN having the same level of outcomequality. In turn, the reduced number of nodes may result in a quickerprocessing time, less computational power used, or less memory used,thus saving computing time and cost.

Example embodiments described include, but are not limited to thefollowing.

Example 1 may be a system comprising: a higher order neural networkimplemented by one or more computer processors, the higher order neuralnetwork is configured to receive an input, to process the input and togenerate an output; wherein the higher order neural network has aplurality of layers including an input layer, one or more hidden layers,and an output layer; wherein each of the input layer, the one or morehidden layers, and the output layer includes a plurality of neurons;wherein the plurality of neurons includes at least first order neuronsand second order neurons; and wherein inputs at a second order neuronare combined using a symmetric tensor.

Example 2 may include the system of example 1, or of any other exampleherein, wherein the symmetric tensor is a symmetric matrix, thesymmetric matrix combining a plurality of input and a plurality ofcorresponding transposed inputs.

Example 3 may include the system of example to, or of any other exampleherein, wherein the inputs are one dimensional, the symmetric tensor isa quadratic tensor, and the symmetric matrix is a 2×2 matrix.

Example 4 may include the system of example 3, or of any other exampleherein, wherein symmetric matrix represents a selected one of anellipse, a hyperbola, a parabola, or a plane.

Example 5 may include the system of example 3, or of any other exampleherein, wherein the output layer is created using one quadraticperception.

Example 6 may include the system of example 2, or of any other exampleherein, wherein the symmetric tensor is a cubic tensor.

Example 7 may include the system of example 1, or of any other exampleherein, wherein the higher order neural network is trained using agradient descendant algorithm, and an activation function.

Example 8 may include the system of example 7, or of any other exampleherein, wherein the higher order neural network is further trained usingan error function to generate a measurement of amount of error for agiven input, the amount of error representing a degree of differences ofan output generated by the higher order neural network from acorresponding desired output.

Example 9 may be a method for managing a higher order neural network,comprising: identifying inputs for a second order neuron; combining theinputs at the second order neuron with a symmetric tensor; andoutputting the combined inputs.

Example 10 may include the method of example 9, or of any other exampleherein, wherein the symmetric tensor is a symmetric matrix.

Example 11 may include the method of example 9, or of any other exampleherein, wherein combining the inputs further comprises combining aplurality of input and a plurality of corresponding transposed inputs.

Example 12 may include the method of example 11, or of any other exampleherein, wherein the inputs are one dimensional, the symmetric tensor isa quadratic tensor, and the symmetric matrix is a 2×2 matrix.

Example 13 may include the method of example 9, or of any other exampleherein, wherein symmetric matrix represents a selected one of anellipse, a hyperbola, a parabola, or a plane.

Example 14 may include the method of example 9, or of any other exampleherein, wherein outputting the combined inputs further comprisesoutputting the combined inputs to an output layer.

Example 15 may include the method of example 14, or of any other exampleherein, wherein the output layer is created using one quadraticperception.

Example 16 may include the method of example 9, or of any other exampleherein, further comprising training the higher order neural networkusing a gradient descendant algorithm and an activation function.

Example 17 may include the method of example 9, or of any other exampleherein, further comprising training the higher order neural networkusing an error function to generate a measurement of an amount of errorfor a given input, the amount of error representing a degree ofdifferences of an output generated by the higher order neural networkfrom a corresponding desired output.

Example 18 may include the method of example 9, or of any other exampleherein, wherein the symmetric tensor is a cubic tensor.

Example 19 may include the method of example 9, or of any other exampleherein, wherein the symmetric tensor is a quartic tensor.

Example 20 may be one or more computer-readable media comprisinginstructions the cause a computing device, in response to execution ofthe instructions by the computing device, to: identify inputs for asecond order neuron; combine the inputs at the second order neuron witha symmetric tensor; and output the combined inputs.

Example 21 may include the computer-readable media of example 20,wherein the symmetric tensor is a symmetric matrix.

Example 22 may include the computer-readable media of example 20,wherein combining the inputs further comprises combining a plurality ofinput and a plurality of corresponding transposed inputs.

Example 23 may include the computer-readable media of example 22,wherein the inputs are one dimensional, the symmetric tensor is aquadratic tensor, and the symmetric matrix is a 2×2 matrix.

Example 24 may include the computer-readable media of example 20,wherein symmetric matrix represents a selected one of an ellipse, ahyperbola, a parabola, or a plane.

Example 25 may include the computer-readable media of example 20,wherein outputting the combined inputs further comprises outputting thecombined inputs to an output layer.

Example 26 may be an apparatus for managing a higher order neuralnetwork, comprising: means for identifying inputs for a second orderneuron; means for combining the inputs at the second order neuron with asymmetric tensor; and means for outputting the combined inputs.

Example 27 may include the apparatus of example 26, or of any otherexample herein, wherein the symmetric tensor is a symmetric matrix.

Example 28 may include the apparatus of example 26, or of any otherexample herein, wherein means for combining the inputs further comprisesmeans for combining a plurality of input and a plurality ofcorresponding transposed inputs.

Example 29 may include the apparatus of example 28, or of any otherexample herein, wherein the inputs are one dimensional, the symmetrictensor is a quadratic tensor, and the symmetric matrix is a 2×2 matrix.

Example 30 may include the apparatus of example 26, or of any otherexample herein, wherein symmetric matrix represents a selected one of anellipse, a hyperbola, a parabola, or a plane.

Example 31 may include the apparatus of example 26, or of any otherexample herein, wherein means for outputting the combined inputs furthercomprises means for outputting the combined inputs to an output layer.

Example 32 may include the apparatus of example 31, or of any otherexample herein, wherein the output layer is created using one quadraticperception.

Example 33 may include the apparatus of example 26, or of any otherexample herein, further comprising means for training the higher orderneural network using a gradient descendant algorithm and an activationfunction.

Example 34 may include the apparatus of example 26, or of any otherexample herein, further comprising means for training the higher orderneural network using an error function to generate a measurement of anamount of error for a given input, the amount of error representing adegree of differences of an output generated by the higher order neuralnetwork from a corresponding desired output.

Example 35 may include the apparatus of example 26, or of any otherexample herein, wherein the symmetric tensor is a cubic tensor.

Example 36 may include the apparatus of example 26, or of any otherexample herein, wherein the symmetric tensor is a quartic tensor.

Although certain embodiments have been illustrated and described hereinfor purposes of description, a wide variety of alternate and/orequivalent embodiments or implementations calculated to achieve the samepurposes may be substituted for the embodiments shown and describedwithout departing from the scope of the present disclosure. Thisapplication is intended to cover any adaptations or variations of theembodiments discussed herein.

Where the disclosure recites “a” or “a first” element or the equivalentthereof, such disclosure includes one or more such elements, neitherrequiring nor excluding two or more such elements. Further, ordinalindicators (e.g., first, second or third) for identified elements areused to distinguish between the elements, and do not indicate or imply arequired or limited number of such elements, nor do they indicate aparticular position or order of such elements unless otherwisespecifically stated.

What is claimed is:
 1. A system comprising: a higher order neuralnetwork implemented by one or more computer processors, wherein thehigher order neural network is configured to receive an input, toprocess the input and to generate an output; wherein the higher orderneural network has a plurality of layers including an input layer, oneor more hidden layers, and an output layer; wherein each of the inputlayer, the one or more hidden layers, and the output layer includes aplurality of neurons; wherein the plurality of neurons includes at leastfirst order neurons and second order neurons; and wherein inputs at asecond order neuron are combined using a symmetric tensor.
 2. The systemof claim 1, wherein the symmetric tensor is a symmetric matrix, thesymmetric matrix combining a plurality of input and a plurality ofcorresponding transposed inputs.
 3. The system of claim 2, wherein theinputs are one dimensional, the symmetric tensor is a quadratic tensor,and the symmetric matrix is a 2×2 matrix.
 4. The system of claim 3,wherein the symmetric matrix represents a selected one of an ellipse, ahyperbola, a parabola, or a plane.
 5. The system of claim 3, wherein theoutput layer is created using one quadratic perception.
 6. The system ofclaim 2, wherein the symmetric tensor is a cubic tensor.
 7. The systemof claim 1, wherein the higher order neural network is trained using agradient descendant algorithm, and an activation function.
 8. The systemof claim 7, wherein the higher order neural network is further trainedusing an error function to generate a measurement of amount of error fora given input, the amount of error representing a degree of differencesof an output generated by the higher order neural network from acorresponding desired output.
 9. One or more non-transitorycomputer-readable media comprising instructions that cause a higherorder neural network implemented by one or more computer processors, inresponse to execution of the instructions by the one or more computerprocessors, to: identify inputs for a second order neuron; combine theinputs at the second order neuron with a symmetric tensor; and outputthe combined inputs.
 10. The non-transitory computer-readable media ofclaim 9, wherein the symmetric tensor is a symmetric matrix.
 11. Thenon-transitory computer-readable media of claim 9, wherein combining theinputs further comprises combining a plurality of input and a pluralityof corresponding transposed inputs.
 12. The non-transitorycomputer-readable media of claim 11, wherein the inputs are onedimensional, the symmetric tensor is a quadratic tensor, and a symmetricmatrix is a 2×2 matrix.
 13. The non-transitory computer-readable mediaof claim 9, wherein a symmetric matrix represents a selected one of anellipse, a hyperbola, a parabola, or a plane.
 14. The non-transitorycomputer-readable media of claim 9, wherein outputting the combinedinputs further comprises outputting the combined inputs to an outputlayer.